^  ^  COPY  WA1049 38 


*1 


'I'li XA  S  A&M  UNI  VKKSITV 

t.OILKiL  SIAIION,  IIXAS  77»4 i 


/ 


Ismi  11  11  111  Si  misiii  \ 
I’lmiir  TH  -  MS-1141 


ON  SOM  1C  NUMKRIOAJ.  PROPKRT  I  ICS  OF 
ARMA  PARAMETER  ESTIMATION  PROCEDURES 


by  H.  Joseph  Newton 
Institute  of  Statistics 
Texas  A&M  University 


Technical  Report  No.  N-25 
July  1981 


Texas  A  &  M  Research  Foundation 
Project  No.  4226T 

"Robust  Statistical  Data  Analysis  and  Modeling" 

Sponsored  by  the  Office  of  Naval  Research 
Professor  Emanuel  Parzen,  Principal  Investigator 

Approved  for^public  release;  distribution  unlimited. 

* 

t 

6  i  9  3  0  0  36 


DTiC 

ELECTC 
OCT  1  1981 


D 


t* 


Uni- 1  ass  I  f  I « m I 

m  cum  i  v  cl  asm i  it  a i ion  or  nir,  paot  r »?.«.,  nm  r.nimmi) 

I  REPORT  DOCUMENTATION  PAGE 


I  .  \ - .  1  nr  ixm^t  miimi. 


7.  GOVT  ACCESSION  NO. 

’  A lof g?, 


T  I  ftr  (ond  Lifilllla) 


On  Sonic  Nnnicr  I c;i  I  Properties  of  ARMA  Parameter 
t\N I  limit  Ion  Procedures, 


|T.  AUTHOR^.) 


I  T  H.  Joseph  Newton 


*  PERFORMING  ORGANIZATION  NAME  ANO  ADDRESS 

Tows  A&M  University 
Institute  of  St  at  1st ies . 

College  St.it  ion,  TX  7 7 S A  1 


II  CONTROL  l.ING  OF  FICE  NAME  AND  ADDRESS  / 

U.  S.  Anny  Research  Office  \ 

I’.  0.  Box  12277 

Research  Triangle  Park.  NC  27709 _ 

T«  MONITORING  AGENCY  NAME  »  ADDRESSrlf  dlltormnl  from  Controlling  Otllco) 

,  ,  1  9 


|i»  DISTRIBUTION  STATEMENT  (ol  Ihl.  Hoporl) 


BEAD  INSTRUCTIONS 
BEFORE  COMPLETING  FORM 
S.  RECIPIENT'S  CATALOG  NUMBER 


S.  TYPE  OF  REPORT  •  PERIOD  COVERED 

/  J  '  Technical  r -  j 

_  i~ _ 

s.  PERFORMING  ORG.  REPORT  NUMBER 


S.  CONTRACT  OR  ORANT  NUMBERS 

!  D AAG2  9  -  8^  C-Jifyf 7  0 


10.  program  element,  project,  task 
AREA  4  WORK  UNIT  NUMRERS 


1 1 1  JuH—  81  _ 

T*»<niMBtW  of  Vases 

6  (six) _ 

IS  SECURITY  CLASS,  (ol  IMr  npMIJ 

Unclassified 


ISa.  DECLASSI  FI  CATION/ DOWN  GRADING 
SCHEDULE 


Approved  for  public  release;  distribution  unlimited. 


[  17.  DISTRIBUTION  STATEMENT  (of  the  abetted  anf«r<rf  In  Block  20,  II  dllterent  Item  Report) 


I  14  SUPPLEMENTARY  NOTES 


19  KEY  WORDS  (Continue  on  reverae  aide  It  naceeeery  and  Identify  by  block  number) 

ARMA  models.  Maximum  likelihood  estimation.  Minimum  mean  square  error 
prediction,  Kalman  filter  algorithm,  Modified  Cholesky  decomposition  algorithm. 


To  A**S1  RACT  fContlnao  on  rmrttmm  ml  do  II  NtlMlfi  mu)  Identity  »F  SlecA  numb")  "This  paper  revleWS  the  algOT  thlHS 

used  by  statisticians  for  obtaining  efficient  estimators  of  the  parameters  of  a 
univariate  autoregressive  moving  average  (ARMA)  time  series.  The  connection  of 
the  estimation  problem  with  the  problem  of  prediction  is  investigated  with 
particular  emphasis  on  the  Kalman  filter  and  modified  Cholesky  decomposition  ^ 

algorithms.  A  result  from  prediction  theory  is  given  which  provides  a  signlfic  inf. 
redact  Ion  In  the  consultations  needed  In  Ansley's  (1979)  estimation  procedure. 

Finally  it  Is  pointed  out  that  there  arc  many  useful  facts  in  the  literature  of 
control  theory  that  need  to  be  investigated  by  statisticians  interested  in 


S/N  0 1  OJvLF*  0 1 4*  660 1 

*o 


fvblaws  in  linear  time  series  models. 

7  .  Unclassified  _ 

•ECURITT  CLASSIFICATION  OF  THIS  PAOC  |WRm  Om  M 


ON  SOME  NUMERICAL  PROPERTIES  OF  ARMA  PARAMETER  ESTIMATION  PROCEDURES 


II.  Joseph  Newton 
Institute  of  Statistics, 
Texas  A&M  University 


Abstract 

This  paper  reviews  the  algorithms  used  by  ntnt  Istlcians  for 
obtaining  efficient  estimators  of  the  parameters  of  a  univari¬ 
ate  autoregressive  moving  average  (ARMA)  time  series.  The 
connection  of  the  estimation  problem  with  the  problem  of  pre¬ 
diction  Is  investigated  with  particular  emphasis  on  the  Kalman 
filter  and  modified  Cholesky  decomposition  algorithms.  A 
result  from  prediction  theory  Is  given  which  provides  a  signi¬ 
ficant  reduction  in  the  computations  needed  in  Ansley's  (1979) 
estimation  procedure.  Finally  it  is  pointed  out  that  there  are 
many  useful  facts  In  the  literature  of  control  theory  that  need 
to  be  investigated  by  statisticians  Interested  in  estimation 
and  prediction  problems  in  linear  time  series  models. 


Key  Words 

ARMA  models,  Maximum  likelihood  estimation.  Minimum  mean  square 
error  prediction,  Kalman  filter  algorithm.  Modified  Cholesky 
decomposition  algorithm 


1.  INTRODUCTION 


nite  order  autoregressive  process. 


Let  {e(t),teZ),  Z  che  set  of  integers,  be  a 
white  noise  time  series  of  uncorrelated  zero 
mean  random  variables  having  common  variance 
a2.  Then  the  time  series  (Y(t),tcZ)  satisfy¬ 
ing 

P  9 

Y (t)  -  -l  a(J)Y(t-j)+c(t)+  l  8(k)e(t-k)  (1.1) 

J«1  k«l 

is  called  an  autoregressive  moving  average  pro¬ 
cess  of  order  p  and  q  (ARMA(p,q)).  If  p=0  then 
Y(-)  is  a  moving  average  process  of  order  q, 
MA(q),  while  if  q-0  Y(*)  is  an  autoregressive 
process  of  order  p,  AR(p). 

Defining  a(0)-B(0)“l  and  the  complex  valued  poly¬ 
nomials 

P  a  9  fc 

g(z)«  Z  a(J)zJ  ,  h(z)-  Z  8(k)z  , 

J-0  k-0 

we  can  write  (1.1)  as 

P  9 

l  a(J)Y(t-J)“  Z  B(k)e(t-k) 

J-0  k-0 

or 

ga)Y(t)-h(L)c(t), 

where  L  is  the  lag  or  back  shift  operator, 

L^Y(t)“Y(t-J) ,  JcZ.  If  the  zeros  of  g(*)  are 
outside  the  unit  circle  then  Y(-)  is  stationary, 
l.e.  it  can  be  written  as  an  infinite  order 
moving  average  process,  while  if  the  zeros  of 
h(-)  are  outside  the  unit  circle  then  Y(-)  Is 
Invertible,  l.e.  it  can  be  written  as  an  lnfi- 


The  ARMA  model  has  been  very  useful  in  analy¬ 
zing  time  series  data.  Given  a  sample  reali¬ 
zation  YT=(y(l) , . . . ,Y(T))  from  Y(-)  one  seeks 
T  .  T  .  . 

estimators  o-(a(l) , . . . ,a(p) )  ,  g” (0(1> . . . . 

6(q))‘,  and  oz  of  the  parameters  a,g,  and  o2, 

as  well  as  memory-t,  horizon-h,  minimum  mean 

square  error  linear  predictors  and  prediction 

variances  Y(t+h|t),  o2  -E{Y(t+h)-Y(t+h| t) }2 
c  t  n 

of  Y(t+h)  given  Y(l) , . . .  ,Y(t)  for  a  variety  of 
memories  and  horizons  t-t^, . . . ,t^t  h-h^, . . . .hj. 

Thus  Y(t+h| t)-X?  Y  where  X.  .  minimizes 

"Li  U*t  "*L  | n 

S(£)»E(Y(t+h)-iTYt}2  , 

and  o2  -S(X  .  ) .  This  gives  (see  Whittle  (1963) 

t,h  -t,h 

p.47)  that  X  ,  satisfies  the  normal  equations 
-t,h 


rY,t-t,h"-t,h  ’ 
while 

°t,h“RY(0)_Jt,h  rY^t,h’ 

where  r^  h“ (R^ (t+h-1) , . . . ,Ry(h))  with  Ry(v)- 

E(Y(t)Y(t+v)),vcZ,  and  Tv  is  the  (t*t)  Toeplltz 

covariance  matrix  of  Y„\  i.e.  T„  .  has  (J,k)th 

I  f  t 

element  Ry(k-J).  We  can  thus  define  t  by  its 
first  row;  Ty  ^TOEPLOty (0) . Ry(t-1)). 


The  usual  assumption  made  for  doing  estimation 
and  prediction  in  ARMA  models  is  that  c(*)  (and 
thus  Y('))  is  a  Cauaalan  process.  Thus  the  maxi- 


«mm  likelihood  estimators  a, (5,  and  o'1  maximize 
the  Gaiianlnn  like!  llim id 

L(o,B.o2/YT)-(?»)'T/2|rYjTr,iexp(-‘iYjr^TYT), 

while  Y (t+h| t)  and  o2  .  are  the  conditional  mean 
c.n 

end  variance  of  Y(t+h)  given  Y (1) , . . . ,Y(t) . 

In  tlila  paper  we_l)  review  nttemptn  liy  statisti¬ 
cians  to  obtain  a,|,  and  a2  (or  estimators 

asymptotically  equivalent  to  them),  2)  show  how 
recent  methods  are  closely  connected  with  finding 
predictors,  and  3)  propose  an  Improvement  of 
Ansley's  (1979)  estimation  procedure.  This  pro¬ 
cedure  Is  currently  regarded,  at  least  in  many 
situations,  as  the  most  numerically  efficient 
available.  Finally  we  will  bo  able  to  compare 
the  two  most  popular  methods  available. 

2.  APPROXIMATE  METHODS 

There  have  been  three  basic  types  of  procedures 
used  to  estimate  the  parameters  of  the  ARMA  model: 
1)  estimators  that  are  derived  heuristlcally  and 
then  shown  to  be  asymptotically  equivalent  to  the 
MLE,  2)  estimators  obtained  by  maximizing  a  func¬ 
tion  asymptotically  equivalent  to  the  likelihood 
L,  and  3)  directly  maximizing  L.  The  procedures 
have  evolved  as  both  computing  software  and  hard¬ 
ware  have  Improved. 

As  an  example  of  a  type  1  procedure  we  consider 
Hannan's  (1969)  method.  The  method  consists  of 
1)  finding  consistent  initial  estimators 

and  2)  performing  an  alternating  proced¬ 
ure  of  the  form  >-8^ 1  ^  ^  ,a^ 1  , 

3)  combining  gi*),§^2)  to  obtain  asymptotically 
efficient  8,  6)  0  ■*  a,  and  5)  possibly  iterat¬ 
ing  the  process  by  returning  to  (2)  with  a  re¬ 
placing  a(°). 

The  Initial  estimators  and  are  obtained 

by  noting  the  following  facts  about  the  ARMA 
model: 

P 

1)  l  a(J)R^(J-v)-0  ,  v-q+1, . . .  ,q+p  (2.1) 

j-0 

2)  By  writing 

g(l)Y(t)-h(L)e(t)«X(t),  (2.2) 

we  note  that  X(-)*vMA(q,B.oz),l.‘e.  X(-)  Is  an 
MA(q)  process  with  parameters  B  and  oz.  Thus 

by  the  first  equality  In  (2.2)  we  have 
P 

Vy>"  z  a(j)a(k)MJ4v-k),veZ,  (2.3) 

*  J.k-0  * 

while  from  the  second  equality  we  have 


q-|v| 

R -(v)'.!2  E  B(k)B(k+|v|)  ,  |v|<q 
*  '  k-0 

.  0  ,  | v| >q  . 


Then  defining  the  consistent  sample  autocovarl- 
ances 

Mv>-|  T'iV^(t)Y(t+|v|)  ,  |v|  <  T  , 

t-1 

is  found  by  solving  (2.1)  with  Rj.(v) 
replacing  R^(v),  while  Is  obtained  by  first 

using  and  RjO)  In  (2.3)  to  obtain  con¬ 

sistent  estimators  R^C0)  (0) , . . .  .R^0)  (q)  which 
are  then  used  In  (2.4)  to  get  §(°)  via  Wilson's 
(1969)  or  Bauer's  (195S)  algorithm. 


Steps  2  and  4  of  Hannan’s  method  are  performed 
in  the  frequency  domain  by  noting  from  (2.2) 
that  the  spectral  density  functions  fy(*)« 

fx(-).  and  f£(-)  of  Y('),  X(>)»  and  e(-)  are 

related  by  (since  f£(w)«  oz/2»  >  we[0,2it]) 

|g(eiM)|2fv(w)-  |h(elw)|z  -fx(w),  (2.5) 

Then  we  can  rewrite  (2.5)  as,  dropping  arguments 
for  convenience, 


|g|ZfY  »i,2/o2  1 

2  "  2x  IhP 

X 


(2.6) 


Y  a 2  1 

TiP 


(2n/a2)2|g|zfy  ^2  x 

- fhp - W7  *  (  8) 

Thus  the  right  hand  sides  of  (2. 6), (2. 7),  and 
(2.8)  are  In  the  form  of  an  autoregressive  spect¬ 
ral  density,  and  thus  for  v>0. 


1  ?W  l(i-v)w  A 

1  6  <J>/  —.7—  eU3  v,wdw  -  S  - 

J-0  0  £X  v 

1  aU)f  ThT7  ei(J_v)wdw  -  6  a2  , 


w  4xz 

dw  -  6  — 7- 

v  a* 


1  ThT7  * 
j-0  0  |nf 


(2.10) 


l  B(J)/  rrry~-  e1(J'v)%>dw  -  6  0Z  . 

J-0  0  !hl  v 

Now  fy  can  be  estimated  by  the  perlodogram 

V  -  £ ,  f„  «,<»>•■'”  • 

|v|  <T 

and  since  X(»)  -  MA(q ,  B,  oz)  we  have  that 


(2.11) 


Then  for  example,  In  nn  obvious  notation,  the 
step  a(0)  •*  8^  In  done,  by  solving  (2.9)  with 
fT,  and  f£°^  replacing  g,  fy,  and  f  and 

the  fast  Fourier  transform  used  to  calculate  a 
rectangular  sum  approximation  to  the  Integrals 

needed.  Then  (2.10)  Is  used  for  8^  *  and 

6  -  a  while  (2.11)  Is  used  for  a(l)  -  8(Z). 

Finally  the  combination  of  B^^  and  In  done 

using  an  analogy  with  two-stage  least  squares. 

In  a  pivotal  article,  Akalke  (1973)  noted  that 
Hannan's  method  was  the  same  ns  one  step  In 
directly  maximizing  L  using  the  Newton-Raphson 
procedure  with  nn  approximation  to  the  Hessian 
matrix  of  L.  This  observation  led  many  re¬ 
searchers  to  turn  their  attention  to  other  pos¬ 
sible  approaches  to  directly  maximize  L.  We  will 
consider  two  of  these  In  the  next  section. 

As  an  example  of  a  type  2  estimation  procedure 
we  consider  the  method  of  Box  and  Jenkins  (1970). 

T  -1 

In  large  samples  the  term  exp(-VfYTry  ^.Y^.)  domi¬ 
nates  the  term  Ir^  t|  Thus  Box  and  Jenkins 

suggest  maximizing 

L'(q,  B,  o2|YT)  -  exp(- 

which  can  be  done  la  a  nonlinear  regression 
framework.  Thus  It  can  be  shown  that 
T 

Tt^y1t-t  ■  1  [e(t)l2  (2a2) 

where  [e(t)]  •  E(e(t)|YT,  a,  B).  Thus  the  Box- 

Jenkins  procedure  consists  of  replacing  by 
-Q  in  (2.12)  and  approximating  (c (t) ]  by  back- 
forecasting. 

It  is  generally  agreed  that  in  the  case  of  large 
T  and/or  zeros  of  h(z)  far  from  the  unit  circle 
that  estimation  methods  of  type  1  and  type  2 
give  very  reasonable  results.  However  if  one 
is  faced  with  a  situation  where  the  above  con¬ 
ditions  are  not  satisfied,  then  the  numerical 
properties  of  the  procedures  nullify  the  bene¬ 
fit  of  their  simplicity.  Thus  Iterations  often 
fall  to  converge  In  a  reasonable  time  and  sys¬ 
tems  of  equations  that  must  be  solved  become 
highly  Ill-conditioned.  Thus  there  has  been  a 
need  for  more  numerically  stable,  exact  maxi¬ 
mum  likelihood  methods. 

3.  EXACT  MAXIMUM  LIKELIHOOD  .PROCEDURES 

With  the  advent  of  iterative  nonlinear  optimi¬ 
zation  procedures  which  require  only  a  starting 
value  and  evaluation  of  the  function  to  be 
optimized  for  given  values  of  Its  arguments 
(see  Dennis  and  More  (1977),  for  example),  the 
most  recent  procedures  for  ARMA  parameter  esti¬ 
mation  have  centered  on  evaluating  L  for  given 
values  of  a,  S,  end  o2. 


In  this  section  we  consider  two  such  methods 
for  evaluating  1)  using  the  Kalman  filter 
algorithm  and  7)  using  covnrlnnce  matrix  de¬ 
composition  methods,  particularly  the  Cholesky 
decomposition  algorithm.  The  basic  purpose  of 
these  algorithms  Is  to  obtain  the  one  step  ahead  - 

prediction  errors  e(t)  -  Y(t)  -  Y(t|t  -  1)  and 
prediction  variances  n2  «  o2_^  t  -  1 .  T 

since  one  con  cosily  show  that 

|rv  |  -  no2  yIx"1  y  -  r  . 

Y.T  t_1  t  _T  Y,T.T  t-1  Ot 

To  summarize,  the  two  methods  currently  regarded 
as  most  numerically  efficient  for  maximizing  L 
consist  of  two  stages: 

1)  Find  initial  estimators  as  described  above 
for  Hannan's  method. 

2)  Use  an  Iterative,  derivative  free  nonlinear 
optimization  algorithm  to  find  the  maximum 
likelihood  estimators,  using  either  the 
Kalman  filter  algorithm  (Akalke  (1976), 

Harvey  and  Phillips  (1979),  Jones  (1980), 
Pearlman  (1980),  or  Gardner,  Harvey,  and 
Phillips  (1980),  for  example)  or  the  Cho¬ 
lesky  decomposition  algorithm  (see  Pagano 
and  Parzcn  (1973),  Pagano  (1976),  Phadke 
and  Kedem  (1978),  Ansley  (1979),  Newton 
(1980),  Newton  and  Pagano  (1981),  for  ex¬ 
ample)  to  find  the  e(t)  and  o2  needed  to 
evaluate  L. 

Kalman  Filter  Algorithm 

Consider  the  following  two  equations: 

Yt+j  ■  HtYt  +  GtUt  (State  Equation) 

Xt+^  -  ®t+lYt+l  +  Vt+1  (°b8ervfltion  equation) 

where  the  Y's  are  unobservable  N-vectors,  the 
X' s  are  observable  M-vectors,  H^,  C^,  and  St  are 

known  (NxN),  (NxL) ,  and  (MxN)  matrices,  and  U^, 

V  are  independent  zero  mean  L  and  N  dimensional 

white  noise  series  with  known  covariance  matrices 
Qt  and  R£. 

Then  given  Initial  values  Y^  and  Pq  *  var(Yg) 
one  finds  the  Y^'s  and  P£  »  cov(Yt)  by  the  Kal¬ 
man  filter  algorithm 

-t+1  "  -t+1  “  Vl(St+l-t+l  *  -t+1* 

Pt+l  “  Vl  '  V1V1V1 
Vi  ’  HtV  Vi  ■  VlSI+l 


<(St+lVlSt+l  *  Vl> 


Vl  ’  HtPtHt  +  w? 


A  menber  of  authors  In  the  statistical  lltera- 


turc  h.ivf*  pointed  out  flint  a  'ilmplo  nppl  f  nit  Ion 
of  this  algorithm  to  ARMA  models  ran  be  made  to 
obtain  the  e(t)  and  o?.  *Thun  since 
f  r 

T(t+J)|t+1)  -  Y(t+j|t)  +  gjf. (t+1)  , 

j  -  l,  ....  n  -  1 
P 

I  a(k)  Y (t+m-k 1 1 )  +  g  c  (t+1), 
k-1  " 

>•  J  "  ■ 

J-l 

where  g  -  1  end  g.  -  8<J-1)  +  I  n(k)g.  .  ,J>2 
1  J  k-1  3  K 


and  m  -  max(p,  q+1),  we  can  write  the  equation 
of  state 

?t+l  ’  H  !t  +  ?  e(t+1)  • 
where  gT  -  (gj,  g? . gj ,  »nd 

H  -  f  0  1  0  ...  0  I 

!  0  0  1  ...  0 


0  0  0 
o(ni)  . 


...  n(l) 


where  a(J)  -  0  If  J  >  p. 

Further,  since  Y  (t  +  1 1 1+1)  -  Y(t+1),  we  can  write 
the  observational  equation 

Xt+1  “  $T  Yt  +  1  +  ?t  +  l  * 
where  X(+1  -  Y(t+1),  ST  -  (1,  0,  ...,  0),  and 
is  an  observational  error  rand  on  variable. 

Cholesky  Decomposition  Algorithm 

An  (nxn)  symsetrlc  matrix  An  Is  positive  defi¬ 
nite  If  and  only  If  it  can  be  written 

’  la  „da  „  •  0.2) 

n  A  ,n  A ,n  a  ,n 

where  L.  Is  a  unit  lower  triangular  matrix  and 
A  |fl 

Is  a  diagonal  matrix  with  positive  diagonal 

elements.  This  Is  the  modified  Cholesky  decom¬ 
position  of  A  .  We  note  that  we  can  also  write 

n  T 

the  Cholesky  decomposition  A  *  M,  M,  where 
,  n  A,n  A,n 

M.  ”  L,  D,  but  we  will  consider  exclusively 
A,n  A,n  A,n 

the  modified  decomposition. 

The  decomposition  (3.2)  Is  unique  and  nested, 

da,J*  LAJ*  »a!j  are  the  0*J)  princi¬ 
pal  minors  of  I..  ,  D,  ,  L.  *  ,  and  D,^  respec- 

A#n  A#n  A,n  Atn  r 

lively  for  J  £  n.  Thus  we  can  denote  the  (],k)th 
element  of  Lv  and  Dv  in  the  decomposition  of 

the  (KxK)  covariance  matrix  of  i  time  series 


Y(*)  ns  l.y  .  .  nnd  D  respectively  for  f.  £  K. 

If  Y(*)  Is  a  purely  nondctermlnlstlc  covariance 
stationary  time  aerlee  with  spectral  density 
function  t(*>,  l.e. 

.T* 

a2  -  2x  exp{-r-  /  log  f(w)dw)  >  0  , 

«  Z*  ^ 

then  e(l)  -  Y(l),  while 
t-1 

e(t)  -  Y(t)  -  E  L  ,  .  e(t-k) ,  t  >  2 
k-1  T,t,c-It 

°t  “  DY , t , t  * 

and  In  fact  (Newton  and  Pagano  (1981)) 
t+h-1 


Y(t+h | t) 


k5h  LY,t+h,t+h-kc(t+h_k) 


h-1 

,2  -  r  1.2 


°t,h  ’  k^QLY , t+h , t+h-k°Y , t+h-k , t+h-k 
Further, 


11m  L v  r  r  ,  -  B  (J),  J  >  0 
K-»  Y*K*K“J 


where  the  B„(-)  are  the  coefficients  In  the  MA(») 
representation  of  Y(-).  While  these  facts  ap¬ 
pear  to  be  known  In  the  literature  of  control 
theory  they  do  not  appear  to  exist  In  the  sta¬ 
tistical  literature,  at  least  In  the  general 
form  of  equations  (3.3)  through  (3.6). 

Now  major  simplifications  of  these  equations 
occur  when  Y(-)  Is  an  ARMA  process.  Thus  (Pa¬ 
gano  and  Parzen  (1973),  Pagano  (1976),  Newton 
(1980))  If  Y(.)  Is  an  MA(q)  we  have  that  j  k 

-Oif  | J  — k |  >0  and  thus  L  .-OlfJ-k 

>  0.  Further,  Pagano  and  Parzen  (1973)  state 
In  an  attempt  to  find  Y(t+h|t)  and  (jZ  ^  that  if 

Y(.)  *  ARMA(p,  q,  a,  B>  o2),  and  one  forms  the 

P  *  * 

series  X(t)  -  t  a(J)Y(t-J),  t  >  p,  then 
J-0 

X(p+1) ,  ....  X(T)  Is  a  realization  from  an 
MA(q,  g,  a2)  process  and  one  can  combine  the 
MA(q)  prediction  algorithm  with  an  AR(p)  pre¬ 
diction  algorithm.  Ansley  (1979)  makes  this 
same  transformation  of  Y(*)  to  X(«)  to  get  the 
e(t)  and  a2  necessary  to  evaluate  L. 

The  methods  of  Pagano  and  Parzen  (1973)  and 
Ansley  (1979)  can  be  sunsnarlzed  theoretically 
by  combining  the  equations  (3.3)  through  (3.6) 
above  with  the  following  theorem  due  to  Newton 
and  Pagano  (1981). 

Theorem  • 

Let  Y( •)  *  ARMA(p,  q,  o,  g,  o2)  and  Z(«) 

*  AR(p ,  a,  o2)  with  associated  covariance  matrix 

sequence.  ry>t  -  snd 

°Z,t4.f 


h,t  “  Lz,tLx,t  • 

°y.t  “  °x,t  • 

where  Xt  -  (X(l) . X(t))T  -  L'lJt  has  jth 

element 


X(J)  -  Y(J) 

J-l 


.  J  ■  l 


E  a  (k)Y(j-k)  ,  J  -  2 . . 

■  k-0  J 

P 

l  a(k)Y ( j-k)  .  J  >  p 

k-0 

where  the  <Vj(*)  are  easily  obcalned  by  perform¬ 
ing  the  Levinson  (19/4)  recursion  for  J  -  p-1, 
....  1,  with  ap+100  -  n(k): 


o  (l)  - 


Vi(1)  -  Vi(J+l)Vi(J+1_1) 


1  <  J  <  p. 


1  -  “J+1<J+1> 


^X  l  )  ’  ^  a»l  (J-®) 

’  ,J  m-maxd.J-p,-*-1 
1 

*  Z  <«!_!«-£) RY<Z-B),i,j>l 

£-max(l,i-p)1 


°t,h  nx,t+h-k,t+h-k 


* "  k-0  .  " 

t+h  "j 

X'f-t+h-kLz*t+h’iLx‘£*t+h'k|2  ‘ 

Thus  to  find  the  one  step  ahead  prediction 
errors  nnd  variances  one  need  only  compute  the 
q  nonzero  elements  of  the  successive  rows  of 

until  the  convergence  properties  (1.7)  and  (1.8) 
take  effect.  Further,  one  can  then  also  find 
the  Y(t+h|t)  f rom  these  same  quantities  nnd  the 
e(').  Note  however  that  to  find  o2  ,  one  also 


needs  to  find  the  elements  of  L_ 


These  can 


be  found  simply  by  noting  that  the  upper  (pxp) 

principal  minor  I.  can  be  found  by  Inverting 
—  *P  _j 

the  lower  triangular  matrix  L_  ,  while 

Z,p 

lz  j  k  "  JikiP  • 

where  y(0)  -  1,  y(l),  y(2),  ...  are  the  coef¬ 
ficients  In  the  MA(»)  representation  of  Z ( * ) . 
Further,  the  elements  In  rows  p+1 ,  ...  In  the 
first  (p-1)  columns  of  are  obtained  by 
P 


Z.p+j.k 


-  -  f  o(r)L 


Z,p+J-r,k 


,  1  £  k  <  p. 


q-|l-j| 

^(U-Jl)  -  o2  Z  8  (k)8 (k+|i-J  |  )  , 
k-0 

|l-j|  <  q  l.j  >  p 


10  K  1  £  J  <_  p,  i  >  p,  and  1  -  J  >  q 

or  If  i,J  >  p  and  |i-j|  >  q 


Z.P+J  ,k 


1  <  k  <  p  . 


Thus  the  predictors  and  prediction  variances 

Y(t+h|t)  and  o2  can  be  obtained  using  either 
t  ,h 

the  Kalman  filter  algorithm  or  the  Cholesky  de¬ 
composition  algorithm.  Also  the  convergence 
properties  described  in  the  Cholesky  algorithm 
can  be  Incorporated  into  the  Kalman  filter  al¬ 
gorithm. 


ii®  t*x  K  K-1  ”  ®d)»  J  "  1,  •••,  q  (3.7) 

lia  DX,K,K  ■  °2-  (3-8) 

K-h» 

Thus  e(l)  -  X(l),  while 

“in(q.j-l) 

«(t)  -  X(t)  -  Z  Lj  .  e(t-k) ,  t  >  1 

°t  ’  Dx,t,t  • 

and  In  fact 

p 

Y(t+h|t)  -  X(t+h|t)  -  I  a(j)Y(t+h-j | t)  , 

-  j  j 


h  ■  l . q 


4.  DISCUSSION 

Thus  we  have  seen  that  algorithms  developed 
recently  for  finding  maximum  likelihood  esti¬ 
mators  of  ARMA  process  parameters  have  essenti¬ 
ally  consisted  of  applying  established  algor¬ 
ithms  for  finding  the  minimum  mean  square  error 
linear  one  step  ahead  predictors  and  prediction 
variances.  We  have  shown  how  the  Cholesky  al¬ 
gorithm  can  be  used  to  find  more  than  one  step 
ahead  predictors  nnd  variances  a9  well.  We  note 
that  both  the  Kalman  filter  and  Cholesky  decom¬ 
position  algorithm  can  be  extended  easily  to 
the  multivariate  ARMA  case.  We  also  note  that 
we  have  not  attempted  to  survey  all  the  recent 
work  In  the  ARMA  estimation  area,  but  rather 
have  emphasized  those  algorithms  that  appear  to 
be  most  widely  used  in  the  literature. 

We  consider  next  the  relative  speed  and  stabil¬ 
ity  of  the  Kalman  filter  and  Cholesky  decom¬ 
position  algorithms.  Pearlman  (1980)  shows  that 


the  number  of  operations  needed  to  find  the  e(t) 
and  via  the  Kalman  filter  algorithm  Is  ap¬ 
proximately  T(2p  t  3m  +  3),  with  m  -  mnx(p,q+l), 
while  for  the  Cholesky  algorithm  it  is 
T(p  +  3j(q+-l ) (q>4) ) .  Thus  If  q  >.  5  the  Kalman 
algorithm  Is  faster. 

Now  the  fact  that  the  Kalman  filter  algorithm 
performs  a  number  of  "matrix  squaring"  oper- 

T 

tlons  (for  example  M  P^H  In  (3.1)  has  enused 

many  Investigators  to  question  Its  stability. 
However  a  number  of  authors  have  suggested  meth¬ 
ods  for  avoiding  these  operations  (see  Paige 
(1976)  for  example).  The  Cholesky  decomposition 
Is  well  known  for  Its  numerical  stability  (see 
Wilkinson  (1967)  for  example).  However  more 
work  Is  needed  before  a  final  conclusion  can  be 
made  about  which  of  the  two  methods  Is  to  be 
preferred. 

Finally  we  point  out  that  there  are  a  variety 
of  numerical  methods  for  analyzing  linear  time 
series  models  in  the  work  of  control  theorists 
particularly  In  a  series  of  papers  of  Kallath 
(see  the  references  In  Aasnaes  and  Kallath 
(1973))  that  need  to  be  Incorporated  Into  the 
statistical  literature. 
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